This blog post was originally published on the USC Center on Public Diplomacy’s Blog on March 23, 2020.
It is no secret that measurement presents formidable challenges to the scholars and practitioners of public diplomacy alike. CPD Faculty Fellow Robert Banks drew our analytical attention to the issue around a decade ago in his CPD Perspectives publication. Since then, there have been various attempts, including the reports by the Advisory Commission on Public Diplomacy. Yet, our search for better assessment methods is far from being over. In a recent project on capturing South Korean country image, we have realized that this never-ending search was partially caused by our moving target. As public diplomacy projects change—as our objectives, campaigns, messages, platforms change—we need to update our approaches.
We decided to study South Korea because the country has been actively managing its global outreach. Let it be through its pop culture (or Hallyu) or through development aid, the country has been carving a spot for itself in global public opinion. Most recently, Korea’s healthcare system made the news all over the world due to its efficient management of the new coronavirus outbreak. Moreover, Korea recently started to evaluate its outreach attempts through large-scale surveys administered by the Korean Culture and Information Service. We had a case country that was not only trying to actively reposition itself but also attempting to assess its success.
Our study started with a public opinion survey administered by a market research institute, which used the five dimensions of country images framework, in Japan, Indonesia, France and the U.S. with a total of 4,006 respondents. The survey included 36 Likert-scale items, measuring the functional, normative, cultural, natural and emotional dimensions of the country image; three open-ended questions, asking about the latest recalled news as well as strengths and weakness of South Korea; and additional demographic and baseline questions.
Lesson #1: Public opinion surveys are necessary, but not sufficient
We assumed that survey results were going to provide an incomplete explanation of South Korea’s image. By their nature, surveys capture public opinion at certain intervals—when they are administered. Country images, on the other hand, are dynamic concepts and therefore have to be complemented by additional data. Our results have supported this assumption. Most frequent responses we recorded in our open-ended questions regarding what they recalled about the country included “none,” “nothing,” “don’t remember,” and “don’t know.”
Yet, respondents answered our Likert scale questions—our response rate was over 80 percent. Even though individuals were not able to recall anything about South Korea, they were able to grade the country when prompted by questions based on their vague notions and stereotypes. We realized we needed unprompted data, which brings us to our second lesson:
Lesson #2: Online data has the potential to complement survey data
There is one place where everyone is providing their unsolicited ideas to everyone else: the Internet! We downloaded all tweets that have used either korea or skorea in its body from February 2019 to February 2020. We averaged around 90,000 tweets per month with the highest number being around 135,000 in July 2019. When we looked at the most frequently used hashtags, we have observed that users were discussing various different aspects of Korea: K-Pop, conflict with North Korea, tensions with Japan, economic performance, and the universal health care system.
A highly active community of Twitter users have consistently tweeted about stopping the practice of eating dog meat in South Korea. We have not observed a similar pattern in our survey data—only three out of 4,006 respondents mentioned dog meat whereas we had seven relevant hashtags in our top 20 most frequently used hashtags list. A simple comparison on Google Trends between the most commonly tweeted three categories, namely BTS, EXO and dog meat consumption in South Korea, reveal that BTS received the most interest in searches, EXO about 50 percent of BTS’s interest, and dog meat consumption less than 1 percent of it. Google Trends data helped us differentiate between genuine interest in topics, and hijacking of conversations by ambitious social media users, such as fans or activists.
While dog activists did not find themselves a place in mainstream traditional media, Twitter gave them a platform to come together and to promote their agenda—which led to our third lesson.
Lesson #3: Online data is a must-have, even without survey data
Online data is important, by itself. Individuals interested in certain topics can use digital platforms to come together and to propagate their ideas. Moreover, with over 3.5 billions active users across the world, social media is where people are! In the U.S., 20 percent of adults get their news from social media platforms.
While scraping digital platforms to compensate for lack of rich textual data or of dynamism in survey data, we should not forget that these platforms are vital aspects of public opinion formation. With nearly half a billion tweets sent per day, there is more than enough data to study various aspects of public opinion formation, bringing us to our fourth and last lesson.
Lesson #4: Online data is necessary, but not sufficient
There is probably too much data online—yet, not all of it is relevant or reliable. Throughout our study, we collected over one million tweets. We have observed how fringe issues can change the discourse (e.g. dog meat). We have seen how widely covered events increase attention paid to the country (e.g. BTS performing in Saturday Night Live or the Koreas-United States Summit).
Online data required contextual and theoretical information. For instance, we observed Gimpo as a frequently used keyword on Twitter—Gimpo is both an airport city and operates dog farms. We further had to refer back to various other contextual resources, ranging from Korea’s policies to K-Pop bands’ tour schedules. Our analysis of textual data revealed 69,260 unique hashtags and 360,002 repeated keywords. Providing a coherent narrative of Korea’s country image cannot be based solely on the numbers and requires a theoretical framework to organize these free-floating concepts.
The way forward
A successful measurement process needs to make use of various data sources. Combining survey data on country image perceptions and online data is necessary to get the full picture of a country’s image and to analyze its composition and structure. Such a combination makes sure we get structured data from surveys for further analysis as well as dynamic unprompted textual data from online resources. Researchers then can interpret the story told by data through contextually and culturally-sensitive social science research.
* This research was supported by the Korean StudiesGrant Program of the Academy of Korean Studies (AKS-2019-R42).